https://pulumi.com logo
Join Slack
Powered by
Can I get confirmation that if you have a mapping ...
# python
d
Can I get confirmation that if you have a mapping of values as an attribute of a component resource written in Python and any of the values in that map are an Output value that the entire map is supposed to be delivered to the component as an output? I'm trying not to spam this channel but I am hitting some absurdly difficult to debug and reconstruct issues. I have a component resource written in Python for creating an EC2 security group, and part of the configuration is to build the ingress rules for the group. Putting what I have looked into into a thread because I don't want to drown out other conversations
What works: • Everything works if I am not using a component as part of a package with a generated SDK (i.e. importing the Python library and using the component resource directly works perfectly) • If I don't set any Output values in the configuration for the ingress rules, it also works when used from the packaged component library What doesn't work: • For some reason, when I try to use, for example, a 
pulumi.Output[str]
value for a CIDR block in one of the configuration values for an ingress rule, instead of just setting the cidr block setting as an Output, it appears that the entire ingress_rules setting gets turned into a pulumi.Output So I have code like this defining my component (omitting the pydantic model configs that allow it to accept arbitrary types and all that):
Copy code
import pydantic

...
class EC2SecurityGroupIngressRule(pydantic.BaseModel):
    """Config for an AWS EC2 Security Group ingress rule."""
...

    ipv4_cidr_block: Input[str] | None = None
    """The IPv4 CIDR block this rule applies to."""

class EC2SecurityGroupArgs(pydantic.BaseModel):
    """Config for an AWS EC2 Security Group."""

...
    ingress_rules: Mapping[str, EC2SecurityGroupIngressRule] | None = None
    """The ingress rules for the security group."""
Instead of getting a Mapping, I'm getting a pulumi.Output for 
ingress_rules
and I know this because Pydantic says so:
Copy code
error: custom:index:EC2SecurityGroup resource 'my-group' has a problem: Unexpected <class 'Exception'>: 1 validation error for EC2SecurityGroupArgs                               
    ingress_rules                                                                                                                                                                                   
      Input should be a valid dictionary [type=dict_type, input_value=<pulumi.output.Output object at 0x7ff549a22c40>, input_type=Output]                                                           
        For further information visit <https://errors.pydantic.dev/2.11/v/dict_type>:
...
I have tried recreating this issue using TypedDicts but the amount of work it takes to recreate all of the moving bits is monumental and I don't want to have to give up type safety and model validators to easily set defaults just to be able to write multi-language components.
The place where I am instantiating the component resource looks like this:
Copy code
my_security_group = EC2SecurityGroup(
            f"{resource_id}-something",
            args=EC2SecurityGroupArgs(
                name=f"vpc-{resource_id}-my-endpoint",
                description="Security group for my endpoint",
                ingress_rules={
                    "https": EC2SecurityGroupIngressRuleArgs(
                        name="https",
                        protocol=EC2SecurityGroupRuleProtocol.TCP,
                        from_port=443,
                        to_port=443,
                        ipv4_cidr_block=vpc.ipv4_cidr_block, <--- This is an Output[str]
                        description="Allow HTTPS access",
                    ),

                },
                vpc_id=vpc.vpc_id,
            ),
        )
It's not like I am wrapping this argument in an apply() call or something, so I have absolutely no idea what I could possibly be doing here to even end up in the situation where I have an output for this value. It just doesn't make any sense and I can't see anything super obvious that I have done wrong on my side
m
just glancing at this, i'm not familiar with what your model setup is. But I believe yes when you compile the SDK, containers with Outputs in are lifted to Outputs themselves. I think it does this to ensure everything can be validated within the async model that pulumi follows. I might be misunderstanding your expected output but you mention apply - apply would be unwrapping. i.e. if you validated within an apply you'd get the type you expect. Could this be as simple as your models accepting pulumi's Input[T] type?
this would still be valid for just T
d
So I have a model that has a field on it that has a field which takes a map of an object, and inside that object I have an Input[str] field. If everything is set to str, then the object sent over is a dict, which is what I am expecting to be sent. When I set the field to be an Output[str], what I am expecting is that the field inside the object is an Output[str] but still sent as a dict, but what actually is sent is an Output of (possibly) the whole dict
Having the whole dict get converted to an Output makes handling the data in the component very painful as you potentially have layers of Outputs and the whole component's content might as well live in an apply, and (as far as I understand) you wouldn't be able to see what changes would be made during an update if the whole thing lives in an apply
m
Yeah I do think this is a pain point for some people. I do think in theory you'd not have to expect layers of Outputs, but it depends what you're doing. The most I've read up on it, I believe Pulumi uses its own from_input to "work it out"
d
So I understand that the documentation says that it will ensure that any nested Outputs are provided as unwrapped values, which is nice, but that then suggests to me that you would need to maintain two copies of all models to ensure you can type functions that operate on a structure that, by the letter of the stated type definition, has nested Outputs. For example, in my security group struct if I try to do a for loop over the map items, if I call a function that accepts the object as a parameter, I would need to have a second model that matches the first one but with no Output values marked on it so that it could be in the type definition of the function called inside the apply
Just seems very unusual to me that this Output object needs to be bubbled all the way to the top
m
I have been thinking lately whether there's a way to make a sort of "pulumi friendly" decorator to wrap around pydantic and do some recursive stuff out of site. But agreed it's a little gnarly
all I can advise as a starting point for now though is just to start implementing Input[...] into your models and see how quickly you hit a complexity wall
d
Yeah, I'm trying to not end up having models with the words all and apply littered through the constructor so much that they I wear the a, p, l and y keys out on my keyboard first. I can only imagine what the conversation with colleagues would be with regards to making changes to code that looks like that. The idea of a decorator or even just some kind of helper function that I can use to wrap the verbose calls to Output.apply is worth considering, though.
I think the part where this really hurts is that because my Mapping is being used to create objects, I don't know how I can avoid creating resources within an apply, which makes creating multiple resources inside the component resource difficult from a map not work well, which is what creating component resources is all about. Are there any ways to work around that?
Holy guacamole, I recreated it. It seems like resource transforms break this. I'm going to log an issue with my recreation
6 Views
Open in Slack
PreviousNext
Powered by